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[57] ABSTRACT 

A timing control arrangement that dynamically con- 
trols the distribution of timing information in a distrib- 
uted digital communication system. A reference timing 
signal is distributed from a reference master node to all 
other nodes in the system. The distribution is accom- 
plished on a dynamic basis without the use of a central 
control. Each node is connected by links to at least one 
other node and each node receives timing signals from 
all of the links to which it is connected. Each node 
selects one of these signals as its timing reference by 
scanning the various signals it receives to identify the 
one signal that is applied via a link path that is the ' 'clos- 
est" to the master reference node as indicated by infor- 
mation specifying the number of intermediate nodes 
through which the timing signal has traveled from the 
reference node to reach the receiving node. This deci- 
sion is based on a set of bits included in the timing signal 
transmitted by each node indicating the number of in- 
termediate nodes between itself and the reference node. 

26 Claims, 12 Drawing Sheets 
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DISTRIBUTED TIMING CONTROL FOR A SOLUTION 

DISTRIBUTED DIGITAL COMMUNICATION The above problem is solved and a technical advance 

SYSTEM is achieved by the provisions of improved timing facili- 

5 ties for a multinode digital system. These improved 

FIELD OF THE INVENTION timing facilities do not require the use of a centralized 

-ru* • *: t * * * • . •* i j j * *i i common controller to: 

This invention relates to a distributed digital system m vr™;^ + u 0 „„™«, u^nu „p „ ♦ . u • ■ 

. _ tit* j e -1-^ * , W Monitor the current health of network timing at 

and more particularly to improved facilities for control- ^ n o des . 

ling the timing between the nodes of the digital system. l0 (2) De te Ct troub i e in ^ timing of one or more nodes; 

PROBLEM m * 

_ M , , (3) Reconfigure the network timing signal paths 

It is necessary that all nodes of a distributed digital when trouble is detected so as to maintain the precision 

system, such as a digital switching or transmission sys- timing at all nodes. 

tern, have precisely synchronized timing so that voice 15 The above functions are achieved without the use of 
and/or data signals can be transmitted error free be- a centralized common controller. Instead, a distributed 
tween nodes. One way to achieve this timing precision timing control arrangement is utilized wherein each 
is to transmit timing signals to all nodes from a master node has sufficient intelligence to perform the functions 
signal source so that the timing circuitry at all nodes is characterized in steps 1, 2, and 3 immediately above, 
synchronized to the same signal. For reliability pur- 20 One node is designated as a master in accordance 
poses, it is necessary that the system timing facilities with the present invention and this master node pro- 
have the capability of maintaining the internode syn- vides a timing signal that is propagated to all other 
chronization at all times including circumstances in nodes of the system to provide each node with a timing 
which the master signal source may fail or become reference of the required precision. The master node 
otherwise unavailable. This reliability has heretofore 25 generates a timing priority number (hereinafter TPN), 
been achieved by the provision of a central controller sa ^ a *» w !* ich * transmitted from the master node, 
which is connected to all nodes and which exchanges together with the master node timing signal, to every 
messages with all nodes to: other node t0 which ^ mast er node is directiy con- 

(1) Initialize the network and designate the network mn nect ^ by a distal transmission path. Each digital trans- 
node that is to provide the master signal from which all 30 misston p ^ in tl ! e P resent d * closur « » divided into 
other nodes are to take their timing; f™."?? nme » «*ordance with well known 

(2) Monitor the health of all nodes; dl ™ mU K U f e * ed tec + hm <I Uf *- In °? her wnis, 
f-x\ rw~~* ~ * • , . . each node to which the master node is directly con- 

(3) Detect trouble conditions during which one or nected recdves timi d ^ embedded in the ^ eam 

more nodes may receive either no timing signal whatso- 35 of useful data ^ fr * m g the master node M wd , ^ m 

ever or a timing signal that has become inadequate for TPN of 1. Each node other then the master operates in 

reliable synchronization; such a manner that it increraents the lowest XPN it 

(4) Reconfigure the network timing signal paths receives by one and then sends this incremented TPN to 
when trouble is detected so that all nodes continue to each node to which it is directly connected. Thus, each 
receive a reliable timing signal. In some cases, particu- 40 node to which the master node is directly connected 
larly if the trouble is in the master node, this may re- receives the TPN of 1 from the master node, it incre- 
quire that an alternate master source of timing signals be ments this TPN of 1 by 1 to form a TPN of 2 and it 
specified and activated; and transmits this TPN of 2 to each node to which it is 

(5) Monitor the health of all nodes, via an exchange of directly connected including to the master node. Each 
messages, to ensure that the network reconfiguration is 45 such directly connected node, other then the master, 
successful, that receives the TPN of 2 increments the TPN of 2 by 

The above characterized timing facilities operate 1 and sends a TPN of 3 to each node to which it is 
satisfactory so as to perform their intended functions. directly connected. 

However, these facilities require the use of a centralized Tins process continues so that each node receives a 
intelligent common controller connected to all nodes 50 TPN from one or more otn er nodes, increments the 
and they are complex, they are expensive, and they lowest TPN it receives and transmits its incremented 
require that messages be sent continuously between all TPN t0 each node t0 wnich it is directly connected, 
nodes and the central controller. This requirement for Most nodes in the network are directly connected to 
the continuous exchange of messages is particularly mor u e one node ^ thus ^ receive a TPN from 
disadvantageous since the system is ill equipped to 55 T Qode to ™ hlch U 1S conn e<* ed * In many cases, a 
transmit and process messages when trouble is detected. n 1 ode T V rCCeiVe T PNs ° f differin S v ^es. In this case, 
The reason for this is that the network is already over X *tZ t T^tpw ? C that it identifies the 

burdened with heavy message flow and message pro- 2225* ^ J PN U reives, »t increments only this 
cessing chores to route traffic around faUed nodes even „ 

untii/x.it fu n n jju; AMft i _ • _ . c - 6° nodes. The master node does not perform a TPN mere- 

2 1 of £T a l TT V menting function. Instead the master node always gen- 

the flow of Urning. Additionally, it is desirable in erates a TPN of j an<J ^ sendfi ^ „ * * 

modem networks in. which control of routing and other C ach node to which it is directly connected. The master 

functions is distributed, to provide a distributed means node receives an incremented TPN of 2 back from each 

of controlling timing flow so that a separate centralized 65 node to which it is directly connected. However the 

and architecturally incompatible timing controller is master node does not increment a TPN it receives and 

not needed. This also allows each node to execute the instead continues to send a TPN of 1 to each node to 

same software. which it is directly connected. 
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The node TPNs are transmitted from a sending node and applies its TPN of I to each node to which it is 
to a receiving node over the same path by which data is directly connected. Each such directly connected node 
transmitted between the two nodes. Each node, other receiving the TPN of 1 cannot possibly generate a TPN 

than the master node, operates in such a manner that it of other than 2 as long as the master node remains oper- 
first identifies the path over which the node receives the 5 able since the algorithm by which each node operates is 

lowest TPN and then, it derives the timing signal re- that is increments the lowest TPN it receives by 1 and 

quired by the node from the data signals transmitted transmits this incremented value out to each other node 

over the path that supplies this lowest TPN to the node. to which it is directly connected. 

Each node periodically reads the TPNs it receives In ah analogous manner, the TPN generated by each 
over the paths to which it is connected and re-evaluates 10 remaining node in the network is constrained and re- 
the relative magnitude of the received TPNs to deter- mains constant as long as the master node and all other 
mine that the path over which it is presently receiving nodes remain operable and supply valid data including 
the lowest TPN is the same as the path that priorly constant TPNs. The TPN transmitted by each node is 
supplied the lowest TPN and the selected timing signal determined by the number of intermediate nodes be- 
for the node. This re-evaluation is made periodically 15 tween a particular receiving node and the master node, 
and a node takes no action with respect to the control of Let it now be assumed that the master node or the 
its timing if no change has taken place regarding which paths extending from it to its directly connected nodes 
path was and still continues to supply the lowest TPN becomes inoperable or unreliable. The directly con- 
to the node. On the other hand, if an analysis of the nected nodes now no longer receive a TPN of 1. In this 
currently received TPNs indicates that the path that is 20 case, there is no constraint on each such directly con- 
now supplying the lowest TPN is different from the nected node and it no longer is constrained in its genera- 
path that priorly supplied the lowest TPN, then the tionof a TPN by the TPN of 1 received from the master 
node switches the control of the timing of the node node. The directly connected node that is no longer 
from the prior path to the path that presently supplies receiving a TPN of 1 from the master node now identi- 
the lowest TPN. This situation occurs when the path 25 ties the lowest valued TPN it is presently receiving 
that priorly supplied the lowest TPN is connected to a from one of the other nodes to which it is connected and 
node that becomes defective and begins to generate it increments this new lowest TPN by 1 and transmits 
TPNs of increasing value as subsequently described. this new TPN to the network nodes to which it is di- 

The above described capabilities whereby each node rectly connected. These other nodes in turn increment 

takes its timing from the path that supplies the lowest 30 the new TPN they receive and send the newly incre- 

, TPN ensures that each node is controlled by the timing mented TPN back to a first node which, in turn, incre- 

signal from the master node over the path that involves ments the newly received TPN by 1 and sends it back to 

the least number of intermediate nodes. the other node. 

The node that is to be the master node is designated at This process continues in such a manner that on each 
the time the network is configured and initialized and 35 successive interchange of TPNs between a pair of 
the master node supplies the timing to all nodes by nodes, the TPN associated with a node is incremented 
means of the timing signals embedded in the data signals by two. The backup master node detects this phenome- 
..that are transmitted from the master node to each node non as it receives successively higher valued TPNs. 
to which it is directly connected and from there to all Finally, when the lowest valued TPN reaches a certain 
other nodes. At the time the network is configured, one 40 predetermined number, such as for example 30, the 
of the other nodes of the network is designated as the backup master determines that the timing of the net- 
backup master node. This backup master functions in work is unstable and it assumes the roll of a master 
the same manner as all the regaining nodes as long as node. It does this by adjusting its circuitry so that it 
the master node is operative and generates reliable tim- immediately begins to send a TPN of 1 to each node to 
ing signals. However, the backup master node supplies 45 which it is directly connected. This causes the timing of 
the network timing in the event the master node fails or the network to become stable and to be controlled by 
in the event that the interconnections between the mas- the timing signals from the backup master node, 
ter node and the remainder of the network is severed or The backup master node remains in charge of the 
otherwise becomes unreliable. The backup master node network until it is advised that the master node is once 
is programmed with information specifying that it is a 50 again operable and at that time the backup master node 
backup master and when it detects that the master nodes ceases to generate a TPN of 1. The timing of the net- 
has failed, it takes charge of network timing by sending work is again brought under control of the master node 
out a TPN of 1 over each network path by which it is which is now operable and it again supplies a TPN of 1 
directly connected to another node. to the network nodes to which its is directly connected 

The backup master determines that the master node 55 in the same manner as priorly described, 

has failed by repeatedly analy2ing the TPNs it receives In summary of the above, it can be seen that the 

and by determining that the TPNs it receives are incre- present invention solves the above discussed problem 

mented on each successive analysis. The decision that and achieves a technical advance by providing timing 

the master node has failed is made when the received facilities which permit a multi-node network's timing to 

TPNs reach a certain predetermined value. As already 60 be accurately controlled at all times either by a master 

mentioned, the master node continuously generates a node or by a backup node without the necessity of a 

TPN of 1 and transmits this 1 to each node to which it centralized common controller which must continu- 

is directly connected. This TPN of 1 controls each node ously burden the network with the exchange of mes- 

rcceiving this TPN in such a manner that the TPN sages between it and the other network nodes. Thus, the 

generated by each such receiving node is incremented 65 disclosed invention permits the timing of the network to 

by 1 and becomes a two. Thus, each node that is di- be controlled with precision in a manner that does not 

rectly connected to the master node can only generate involve the cost or complexity of a centralized control- 

a TPN of 2 as long as the master node remains operable ler. 
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BRIEF DESCRIPTION OF THE DRAWINGS vented from supplying timing back to itself through the 

network, which would create timing loops and network 

This invention may be better understood by a reading timing instability, 

of the following detailed description thereof with refer- The system timing remains in the state portrayed by 

ence to the following drawings in which: 5 FIG. 1 as long as all elements of the network, including 

FIGS. 1 and 2 illustrate the network topology of one the master node in path 101 remain operable and supply 

possible network having a plurality of nodes; the required timing signals to node B. Now, let it be 

FIGS. 3, 4, 5 and 6, when arranged and shown in assumed that either master node A or path 101 encoun- 

FIG. 7, disclose the details of the circuitry provided at ters problems so that node B no longer receives a TPN 

a node; 10 of 1 from node A. In this case, a TPN of 3 is the lowest 

FIG. 8 discloses the protocol used to transmit the valued TPN received by node B. Node B increments 3 

TPNs over the network paths; by 1 and generates a TPN of 4 which is transmitted to 

FIG, 9 discloses the relationship between the system node C as well as node D. Each of nodes C and D 

clock timing and synchronization facilities and the facil- increment the received TPN of 4 and transmit a new 

ities used to control and generate the TPN at each node; 15 TPN of 5 back to node B. Node B, in turn, increments 

FIG. 10 illustrates the manner in which the TPN 5 by 1 and returns a new TPN of 6 to nodes C and D. 

generated by node can vary under certain system condi- This process continues with respect to nodes B, C and 

tions; and D so that successively higher valued TPNs are gener- 

FIGS. 11 through 17 illustrate in flowchart form the ated. The remaining nodes of the network follow in an 

manner in which the circuitry at each node operates to 20 analogous manner and they also generate successively 

generate a TPN in response to the various system condi- higher valued TPNs. Backup node F receives these 

tions and states that may be encountered by the node. successively higher valued TPNs and when a TPN of a 

DETAILED DESCRIPTION f^S^T^ ^ { \ T * C ™ ed M for exam P le f he 

TPN of 30, node F determines that the network timing 

The network topology is shown in FIG. 1 where the 25 is no longer reliable and it assumes control of the net- 
network comprises the plurality of nodes A, B, C, D, E, work timing by changing its operation so that it begins 
F, G, H and I. The nodes are interconnected by the to generate a TPN of 1 which is transmitted from node 
indicated paths with the numbered arrows. Associated F over paths 107, 106 and 112 to each of nodes C, D and 
with each path is a TPN number that is transmitted over G. 

the path in each direction. The number adjacent to each 30 This condition is portrayed in FIG. 2. The network 

node indicates the TPN that is generated by the node timing soon establishes and the nodes of the network of 

and transmitted on all of its outgoing links. In FIG. 1, FIG. 2 generate and exchange TPNs over the various 

node A is the master node, node F is the backup master network paths as indicated by the numbered arrows and 

and the network conditions portrayed in FIG. 1 indi- the numbers associated with each node on FIG. 2. The 

cate that the master node is functioning properly to 35 backup master remains in control until such time as 

supply timing signals of the required precision to the master node A and/or the path 101 is fixed and again 

rest of the network. Master node A generates a TPN of capable of providing reliable timing signals to node B. 

1 and transmits it to node B as indicated by the arrow The master node A resumes its generation of a TPN of 

pointing in the direction of node B over path 101. Node 1 which is again transmitted to node B. Then, backup 

B receives this TPN of 1, it increments this I by 1 and 40 master node F is advised of this condition and it termi- 

applies a TPN of 2 oyer paths 101, 102 and 103 to nodes nates its generation of a TPN of 1. The network is once 

A, D and C, respectively. Node C receives the TPN of again under control of master node A. This switchover 

2, it increments the 2 by 1 and transmits a TPN of 3 to is accomplished in a manner that both master and 

node D over path 104, Node D receives a TPN of 2 backup master are outputting l*s and supplying network 

from node B and a TPN of 3 from node C. It also re- 45 timing simultaneously very briefly. This is because the 

ceives a TPN of 4 from each of nodes E and F. The condition of having no master temporarily is more 

TPN of 2 is the lowest TPN received by node D and likely to cause network timing instability and more 

therefore it increments the 2 by 1 and transmits a TPN transmission errors than having two masters temporar- 

of 3 to each of nodes B, C, E and F. ily. 

Nodes F, G, H and I work in an analogous manner 50 FIGS. 3, 4, 5 and 6, when arranged as shown on FIG. 
and receive the indicated TPNs. Each such node incre- 7, disclose further details of the equipments comprising 
raents the lowest TPN it receives by 1 and transmits this any of the nodes A through I of FIG. 1 or 2. Each node 
incremented TPN to each node to which it is directly contains the equipments required to receive and trans- 
connected. Thus, node I receives a TPN of 5 as well as mit information over any of the links or paths to which 
a TPN of 6. Since 5 is the lowest value of the two TPNs, 55 it is connected. This information comprises the TPNs as 
node I increments 5 by 1 and transmits a TPN of 6 to already described as well as data representing the infor- 
each of nodes G and H. mation that is transmitted over the network from a 

The circuitry of each node operates in such a manner sending node to a receiving node via any intermediate 

that the timing of the node is controlled, from the signals nodes that may be in the network between the sending 

on the same path that transmits the lowest valued TPN 60 and receiving nodes. 

to the node. Thus, node D receives TPNs of 2, 3, and 4. FIG. 8 discloses the manner in which information is 

Since 2 is the lowest received TPN, node D uses the transmitted over the paths of FIGS. 1 and 2 between 

signals on path 102 to control its timing. In the same nodes. The information is in timed division multiplexed 

manner, node I uses the signals on path 110 to control its form and is transmitted in cyclicly recurring frames 

timing rather than the signals on path 111. Using this 65 with each frame comprising a plurality of timeslots. In 

algorithm, each node receives its timing reference signal the partial frame shown on FIG. 8, timeslot 0 contains 

either directly or indirectly from the master over the the TPN information and the other timeslots contain the 

least number of loops possible, Also,, each node is pre- data that is transmitted from a sending to a receiving 
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node. Framing signals divide the signals of each times- registers and timers which operate under microproces- 
lot. Timeslot 0 transmits the TPN information in the sor control to control the clock system 503 so that the 
form of an eight bit word. The first bit is a spare bit that equipments of the node 300 operate under control of a 
is not presently used. The next bit is a complement bit. preferred clock signal in accordance with the present 
Five bits are used to represent the TPN. The final bit is 5 invention. The operation of the node 300 and the node 
used for parity. The complement bit is used to indicate controller 601 is subsequently described in detail in 
whether the TPN number is sent in a complementary or connection with the flowcharts of FIGS. 11 through 17. 
non-complementary form. This capability is used for With respect to link interface 301, conductor R of 
reliability purposes. The TPN can be sent in comple- link LI carries the signals applied to node 300 and these 
mentary form periodically to allow faults which cause 10 signals are applied to receiver 303. These signals are 
one or more bits of the TPN to be stuck at a 0 or a 1 to divided into discrete timeslots as shown in FIG. 8. Re- 
be detected quickly. The framing bit is used in a con- ceiver 303 receives these timeslot signals and applies 
ventional manner for framing purposes and to permit them over its output on path 324 to timeslot extractor 
the receiving circuitry to distinguish the bits of succes- 307. Timeslot extractor 307 extracts the TPN informa- 
sive timeslots. The TPNs could be transmitted in a 15 tion from the bits of timeslot 0 and applies this informa- 
message oriented protocol such as HDLC, but the em- tion over path 328 to incoming TPN register #1 which 
bodiment shown in FIG. 8 is preferred because less is also designated as element 310. This TPN is applied 
processing power and time is required for TPN process- over path 32S from register 310 to the A register 616 
ing and transmission because no complex protocol must (FIG. 6). of node controller 601. Each of registers 
be transmitted 20 616-619 comprises A and B sections. The signal applied 

FIGS. 3, 4, 5 and 6, when arranged or shown in FIG. to receiver 303 is also extended over path 312 to timing 
7, disclose the circuitry comprising a typical node of recovery element 305 which recovers the clock signal 
FIGS. 1 or 2. The node 300 on FIGS. 3 through 6 embedded in the received data on the R path of link LI 
comprise a plurality of line interfaces 301, 302, 401 and This clock signal is extracted and applied over path 326 
402 each of which terminates one of the links or paths to 25 to contact 2 of timing source selector switch 504 of 
which the node is connected. The function of the equip- clock system 503. Recovered clock is also applied over 
ment of a node is to receive information from a con- path 323 to receiver 303. Clock maintenance informa- 
nected link and to apply information to the connected tion is applied over path 331 to the receiver mainte- 
link. The bidirectional links on the left side of FIGS. 3 nance and interface registers 615. Path 331 comprises a 
and 4 are designated LI, L2, L3, and L4. These links 30 maintenance bus and one of its functions is to supply 
comprise the paths of FIGS. 1 and 2 connected to the register 615 with information indicating whether re- 
node portrayed by FIGS. 3 through 6. For example, ceiver 303 is receiving valid data and whether timing 
?n? e ,E° f 1 iS connected to four P aths 102 » recovery element 305 is successful in recovering good 
104, 105 and 106 and therefore, these four paths corre- timing from the signal received by receiver 303 from 
spond to the bidirectional links designated LI through 35 link LI. 

L4 on the left side of FIGS. 3 and 4. Only nodes D and Path 330 extends from the clock signal generator of 
G of FIGS. 1 and 2 are connected to four paths. The the clock system on FIG. 5 to the link interface units on 
remaining nodes are connected to either one path, two FIGS. 3 and 4. On FIG. 3, path 330 connects to timeslot 
paths or three paths. For a node connected to only one extractor 307 and receiver 303. Path 330 carries the 
path, only one of the links LI through L4 would be 40 clock signal that controls the operation of the node and 
connected on FIGS. 3 and 4. Similarly, less than all of therefore, the clock signal on path 330 controls the 
the links LI through L4 on FIGS. 3 and 4 would be receiver 303 and the timeslot extractor 307. The signal 
used for the other nodes of FIGS. 1 and 2 that are con- from the R lead of link LI is clocked into the receiver 
nected to less than four paths. 303 using the recovered clock applied on path 323 and 

In addition to the link interfaces 301, 302, 401 and 45 the incoming data on path 324 is clocked out of the 
402, each node comprises a digital switching or trans- receiver 303 using the clock signals applied on path 330 
mission system 501, a clock system 503, and a node from the clock system 503 of FIG. 5. 
controller 601. The digital switching system or trans- The output of receiver 303 is also extended over path 
mission system 501 comprises the equipments at the 324 and cable 516 to the input of the digital switching or 
node that receives the data or information, other than 50 transmission system 501. This path carried all the times- 
the TPN information, that is transmitted over the net- lot signals on incoming path R of link LI and applies 
work between nodes. If the network of FIG. 1 or 2 them to the digital switching or transmission system 
represents a telephone network, then the element 501 501. Thus, system 501 receives all of the timeslot signals 
would comprise a digital switching system such as a applied to receiver 303 while incoming TPN register 
digital central office or a digital PBX wherein each 55 310 receives only the timeslot 0 signals representing the 
central office or PBX serves a plurality of connected TPN that is transmitted over path LI to the node repre- 
subsenber stations. On the other hand, if the network of sented by the circuitry of FIGS. 3 through 6. 
FIGS. 1 or 2 represents a digital transmission system, Digital Switching or Transmission System 501 ap- 
then element 501 would comprise a digital system serv- plies the data it wishes to transmit to another node to 
ing a plurality of digital transmission facilities. 60 cable 517 and from there to the link interface and link 

Clock system 503 comprises a precision oscillator 505 that is to extend this data to the other node. Let it be 
and a timing source selector switch 504 which permits assumed that the data is to be transmitted over link LI 
the timing of equipments at the node to be referenced to In this case, the data from system 501 is applied to path 
either the local precision oscillator 505 or to a clock 327 where it is inserted into a signal comprising a series 
signal recovered from the data applied to the node over 65 of timeslots as on FIG. 8. Timeslot inserter 308 also 
any of the connected bidirectional links LI through L4. receives from element 311 the TPN that is to be trans- 
The node controller 601 is a microprocessor controlled mitted in timeslot 0. Inserter 308 combines the timeslot 
element having a microprocessor 610 and a plurality of data on path 327 with the TPN on path 329 and applies 
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the resultant signal systems to transmitter 306 for trans- sion oscillator 505 of the node to be the source of timing 
mission over path T of link LI to the node connected to for the node for the time being. The next state of the 
other end of the link. The timeslot inserter receives the system is entitled at the top "TPN Steady at Possibly 
node clock signal on path 330. The clock signal on path New Values". The value of the TPN received on each 
330 is also extended to transmitter 306 to control its 5 path is steady as indicated by the horizontal line associ- 
operation and this clock signal is embedded in its data ated with each TPN and once again the node controller 
stream for recovery at the node which terminates it. selects the lowest valued TPN to identify the path that 
The TPN that is to be transmitted from the node over provides the source of clock signals for the node. In this 
link LI is stored in register 620 and extended over path example, TPN #4 remains the lowest after the fluctua- 
332 to the outgoing TPN register 311 for line interface 10 tions and the recovered timing signal from the link 
301. The outgoing TPN information is inserted into supplying TPN #4 would again be selected as the 
register 311 and applied by this register over path 329 to source of timing for the node. This switch to a new 
timeslot inserter 308. Inserter 308 inserts the outgoing source of timing signals is made a finite interval of time 
TPN into timeslot 0 and inserts the information on path after conditions stabilize as is subsequently described in 
327 into the remainder of the timeslots of each frame 15 connection with the flowcharts of FIGS. 11 through 17. 
generated by link interface 331. The timeslots of each The following describes the operation of the system 
generated frame are applied from inserter 308 to trans- under various conditios with reference to the flow- 
mitter 306 which in turn extends them over path T of charts of FIGS. 11 through 17. The process begins with 
link LI to the node of the network connected to the element 1101 on FIG. 11 when the system is first acti- 
other end of link LI. 20 vated. The system is activated at element 1101 and the 

FIG. 9 portrays the manner in which the clock sys- process continues to element 614 where the system 
tern 503 interacts with the node controller 601 and, in reads the contents of the alarm register 614. This regis- 
particular, the processor controlled portion of the node ter stores the alarms representing the various conditions 
controller. The node controller 601 applies control that may be encountered by the system. The process 
signals over path 901 to control the setting of timing 25 continues to element 1115 where it is determined 
source selector switch 504 to determine the source of whether or not alarm register 614 is set to a state repre- 
timing signals to which the circuitry that is to control senting an alarm. If the answer is no, the process ad- 
the node. This source may either be precision oscillator vances over path 1123 to element 1117. If the answer is 
505 internal to the node, or it may be a clock signal that yes, the process advances over path 1126 to element 
is recovered from the data applied to the node over any 30 1116 in which the system clears the alarms and alarm 
one of the links L1-L4 incoming to the node. The clock register 614. The process then advances over path 1127 
system 503 monitors this operation and returns informa- to element 1117 in connection with which the B portion 
tion over path 902 to the node controller 601 with the of the clear timing source select register 602 is initial- 
returend information representing the state of the sys- ized. This register 602 has an A portion and a B portion 
tern 503 under various conditions that may be encoun- 35 as indicated and the contents of the A portion of the 
tered. In most instances, the information on path 902 register control the setting of the switch 504. In other 
will require no change as to the clock signal that is to words, the A portion of register 602 contains informa- 
control the node. On the other hand, under certain tion identifying the timing source that presently con- 
system conditios or states, the returned information may trols the timing of the node. This register has an A and 
cause the node controller to decide that the signal 40 a B portion whose contents are sometimes compared as 
source currently controlling the node timing is unreli- is subsequently described, 

able and that therefore, the timing control of the node FIG. 11 represents an initialization operation and 
should be switched to a different source of clock sig- during this initialization, element 1117 clears the B por- 
nals. The node controller 601 constantly monitors tion of register 602. The process continues over path 
TPNs coming into the node and applies control signals 45 1128 to element 1118 in which the B portions and the 
on path 901 to control which internal or external source out-of-service bit of the A portions of the TPN registers 
of timing clock system 503 selects as its timing reference 616, 617, 618 and 619 are cleared. The process continues 
signal based on current and recent TPN information to element 1119 in which the TPN stable timer 607 is 
and the maintenance information received on path 902. reset. The TPN stable timer 607 and the subsequently 
The manner in which this is accomplished is described 50 described PLL timer 606 and read TPN 605 timer re- 
in detail in the flowcharts of FIGS. 11 through 17. main in the reset state until explicity started. The pro- 
FIG. 10 portrays the manner in which the TPN gen- cess then contines to element 1120 in which the system 
erated by each node normally remains steady but may resets the PLL timer 606. The process continues to 
vary under certain conditions. The first TPN received element 1121 in which the system then enters the preci- 
by a node is shown on the top line of FIG. 9 and is 55 sion oscillator code into the timing source select register 
designated TPN #1, The next line down is designated 602. This code of the precision oscillator is entered into 
as TPN #2 and so on for the third and fourth lines the A portion of register 602. The entry of this code into 
which are designated TPN #3 and TPN #4. The bot- the A portion of register 602 controls the operation of 
torn line is a time line having three different states. The switch 504 via controller 508 so that switch contact W 
left most state represents a steady state in which the 60 engages contact 1 of the switch to connect the output of 
node controller has analyzed the four incoming TPNs the precision oscillator 505 to the input of the phase 
and has selected the lowest one of them to control the lock loop 506 via path 513. This switches the control of 
timing of the node. The second state of the system that the timing circuitry of the node to the output of the 
may be encountered is termed "Select Precision Oscilla- precision oscillator 505. During initialization and some 
tor". During this state, the TPN received on each path 65 subsequently described failure states, the precision os- 
fluctuates as indicated.. This represents an unstable tim- dilator is selected as the source of timing for all nodes 
ing state of the system and, if the timing becomes suffi- so that the errors rate on the links will be as small as 
ciently unstable, the node controller selects the preci- possible and the system will be able to transmit enough 
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valid TPNs to progress to a synchronized state as subse- now advances over path 1204 to element 1213. The 

quently described. backup register 623 is not set for this node and therefore 

The process of FIG. 11 continues to element 1122 in the process advances over path 1214 to element 1205 

which the system clears the access bit portion 612 of the which determines that the master register 622 is set. The 

external access mode register 611. The state of the ac- 5 process then advances through elements 1207, 1209, 

cess bit is, as subsequently described, determines 1211 and 122S whose functions have already been de- 

whether the node controller examines and responds to scribed. 

the information in the external access node register 611. The process repeats, as already described, through 
. It analyzes the contents of register 611 is the access bit elements 1202, 1213, 1205, 1207, 1209, 1211, and 1225 
612 is set to a L It disregards the state of the register 611 10 which continues to cause a TPN of 1 to be extended out 
if the access bit 612 is a zero. As is subsequently de- over all links of this node to the nodes connected to the 
scribed, the external access mode register 611 is used to other end of links L1-L4. The master node continues 
override the state of switch 626 and to cause the clock sequencing through the same elements as already de- 
system 503 to respond to the information in register 611 scribed as long as it remains in an operable state and in 
rather than respond to the information that is normally 15 charge of the control of the timing of the network, 
provided by the present position of switch 626 and the The following paragraph describes the process fol- 
TPN information. A node normally operates as a master lowed on FIG. 12 for a node that is not the master node, 
node if switch 626 is set to position 624 and as a backup All such nodes are referred to as a link node. The pro- 
master if the switch is set to position 625. The node cess begins on element P at the top of FIG. 12 and 
operates as neither a master or backup master if switch 20 extends over path 1201 to element 1202 which deter- 
626 is set to position 621. mines that the node is a link node and extends the pro- 
The process proceeds over path 1133 to element 1134 cess of over path 1203 to element 1205. Switch 626 of 
in which the mode status register 608 is set to the link this node is not set to its master position and therefore 
mode. The term link mode means that the node does the master register 622 of this node is not set. The pro- 
not, for the time being, operate as a source of timing 25 cess continues from the NO output of element 1205 and 
signals, but, instead, is to have its timing circuitry con- over path 1222 to element 1223 which is designated as 
trolled by a timing signal received from another node. an A within a circle. 

The above described functions on FIG. 11 initialize the The process advances to element A on the top of 

system and prepare it for the following described. FIG. 13. The process extends from element A within 

The process next extends from P on the bottom of 30 the circle to element 1301 which causes the mode status 

FIG. 11 to P on the top of FIG. 12, The process pro- register 608 to be set to the link mode. This causes the 

ceeds over path 1201 to element 1202. This is a decision node controller of the node to operate in its link mode 

element in which a determination is made as to whether in which the timing of the node is controlled by signals 

the mode status register 608 is in its link mode state or in received over the links L1-L4 connected to the node, 

its source mode state. The mode status register 608 is 35 The process then extends over path 1302 to element 

every node is set at the present time to its link mode 1303 which causes the A half of the TPN registers 616 

state since this was done in element 1134 when the through 619 to be read. These registers at this time 

. system was initialized. Therefore, the process extends contain the TPNs that are being received over the in- 

. . over path 1203 to element 1205 which determines coming links L1-L4 from other nodes. 

. whether or not the master register 622 is presently set 40 The process then extends over path 1304 to element 

by switch 626. Let it be assumed for the time being that 1305 which determines whether the complement bit of 

the node that is being described with reference to the the timeslot per FIG. 8 is in a set state. If it is not in a set 

flowchart of FIG. 12 is the master node. Therefore, the state, the process extends over path 1306 to element 

master register 622 is set and the process continues over 1310 which determines the validity of the received 

path 1206 to element 1207 which sets the mode status 45 TPN. If the complement bit is in a set state, the process 

register 608 to its source mode status. This indicates that extends from element 1305 and over path 1307 to ele- 

this mode is to provide the timing signals for the net- raent 1308 which complements the received TPN to 

work. In other words, it is to be the master node. The derive the true TPN. The process then extends over 

process continues over path 1208 to element 1209 in path 1309 to element 1310 which determines the valid- 

which the node TPN register 620 is set to 1. The pro- 50 ity of the received TPN. If the received TPN is valid 

cess continues over path 1210 to element 1211 which the process extends over path 1312 to element 1313 

causes a TPN of one to be transmitted out from this which causes a validity bit of 1 to be written into the A 

node to all other nodes to which this node is connected section of the TPN register associated with this TPN 

durectly via one of its links L1-L4. The process next This is one of registers 616 through 619. The particular 

advances to element 1225 which enters the code of 55 TPN register that is written is that which is associated 

precision oscillator 505 into the A portion of timing with the link on which the TPN now being analyzed is 

source select register 602. This causes switch 504 to received. The process then extends over path 1315 to 

connect the output of oscillator 505 to phase lock loop element 1317. 

506. Clock signal generator 507 then supplies a clock to If element 1310 determines that the received TPN is 
system 501 and the link interfaces 301, 302, 401 and 402 60 not valid, the process extends over path 1311 to element 
which embed the clock in the data they transmit to 1314 which causes the analyzed TPN to be invalidated 
other nodes. This node is therefore now in charge of the A TPN is invalid if it has bad parity, if it is out of the 
network timing. valid range of TPNs, or if its associated out of service 
f m^ P t° CeSS n ° W advances t0 the circIe P at toe top bit is set to a 1 as subsequently described. This is accom- 
2 63 P lished b y writing a validity bit equal to 0 in the A 
The process extends through element 1202 to the section of the TPN register associated with this re- 
Source Mode output of the element since the presently ceived TPN. The process then extends over path 1316 
described node is operating as a master. The process to element 1317 which causes the previously described 
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four steps, beginning with element 1305, to be repeated 
for each of the remaining three received TPNs. Finally, 
after all four received TPNs have been deteremined to 
be valid or invalid, as the case may be, and the appropri- 
ate results entered into the associated TPN registers, the 5 
process extends over path 1319 to element 1320 which is 
designated as a B within a circle. 

The process next extends from the bottom of FIG. 13 
to element B on the top of FIG. 14. At this time, the 
node controller 601 contains in the A section of its four 1° 
TPN registers 606 through 619 the four TPNs that are 
received on the four links L1-L4 incoming to this node. 
This statement presupposes that this node is connected 
to four links and therefore four TPNs are received. If 
this is not the case, then a fewer number of TPNs are 15 
received on the links that are connected to the node and 
these TPNs are stored in the appropriate ones of regis- 
ters 616 through 619. 

The process extends on FIG. 14 from element B to 
element 1401 which causes the node controller 601 to 20 
identify the one of the TPN registers 616 through 619 
that contains the lowest incoming TPN. This lowest 
TPN is identified arid the process extends over path 
1402 to element 1403 which causes the node TPN regis- 
ter 520 to be set to a value that is equal to the lowest 
incoming TPN incremented by one. This number in 
register 620 becomes the TPN of the node. The process 
extends over path 1404 to element 1405 which causes 
the TPN in register 620 to be transmitted over path 332 3Q 
to the outgoing TPN register in each of the link inter- 
faces such as 301. This is register 311 for interface 301. 
The other registers for the other interfaces are registers 
321, 411, and 421. The circuitry of each line interface 
then causes the TPN registered in its outgoing TPN 35 
register to be inserted into its timeslot 0 signal and trans- 
mitted out over the T path of the link L1-L4 to the 
network node connected to the other end of the link. 

The process continues over path 1406 to element 
1407 which enters the contents of the A portion of 40 
registers 616 through 619 into the B portion half of 
these same registers. As is subsequently described, this 
permits a comparison to be made between a priorly 
received and a presently received TPN for each regis- 
ter. The process then proceeds over path 1408 to ele- 45 
ment 1409 which causes the node TPN to be transmit- 
ted out from the outgoing TPN register of each link 
interface out over the associated link to the network 
node connected to the other end of the link. The pro- 
cess then proceeds over path 1411 to element 1412 50 
which is designated F within a circle and from there to 
the top of FIG. 15. 

On the top of FIG. 15, the process extends over path 
1501 to element 1503 which reads the present content of 
all TPN A registers 616-619. The process then extends 55 
to element 1504 which determines whether or not the 
TPN in the node TPN register 620 equals its maximum 
value; What is meant by maximum value is that there is 
a range of acceptable TPN values that the TPNs may 
assume without representing a network trouble condi- 60 
tion. As already mentioned, the master node generates a 
TPN of 1 and each other node in the network generates 
a TPN which is dependent upon the number of nodes it 
is removed from the master node. It has also been men- 
tioned how the TPNs generated by the various nodes 65 
increase if the master becomes inoperable and no longer 
supplies a TPN of 1 to each node which it is directly 
connected. This continued increase in the value of the 
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TPN generated by a node is used in the present inven- 
tion to indicate a trouble condition. 

Let it be assumed that a TPN of 30 generated by a 
node indicates that the node is not receiving valid tim- 
ing information from another node in the network. This 
determination is made by element 1504. If the answer to 
the decision of element 1504 is no, then the node is 
receiving good timing signals and the process extends 
over path 1505 to element 1516 which causes the node 
TPN in register 620 to be outputted to the outgoing 
TPN registers 311, etc. which, in turn, transmit the 
node TPN over their associated links. The process then 
extends to element 1517 which is a G within a circle and 
from there to element G on top of FIG. 16. FIG. 16 is 
subsequently described. 

If element 1504 determines that the value of the TPN 
in the node TPN register 620 has reached a maximum 
value, such as for example 30, the process then extends 
over path 1506 to clement 1507. This element deter- 
mines whether or not the node being described is a 
backup master node. If it is not, then the process extends 
over path 1509 and the node TPN in register 620 is 
transferred to the line interfaces and from there over the 
respective links to the other network nodes. On the 
other hand, if the node being described is a backup 
master node, then the process extends to element 1508 
which causes the switch 504 to apply the output of the 
precision oscillator 505 to the output of the phase 
locked loop 506. The output of the phase locked loop 
506 then is applied to the clock signal generator 507 to 
place the node under control of its own precision oscil- 
lator. 

The process then extends to element 1510 which sets 
the mode status register 608 to source mode to indicate 
that this mode , is now generating the timing signals 
required to control the network. The process next ex- 
tends over path 1511 to elemetn 1512 which causes a 
TPN of 1 to be entered into node TPN register 620. The 
process now extends over path 1513 to element 1514 
which causes the node TPN of 1 to be transmitted out 
via the line interfaces of FIGS. 3 and 4 to the associated 
links and from there to the network nodes to which 
these links are connected. The process then extends 
over path 15 to element 1135 which is designated as a P 
within a circle and is also shown on the top of FIG. 12. 

In partial summary, the process shown on FIG. 15 
has determined that the backup master node is receiving 
invalid timing information and has placed it in charge of 
the network for the time being by cuasing it to begin 
outputting a TPN of 1 to the nodes to which it is con- 
nected. 

From element P on the bottom of FIG. 15, the pro- 
cess now returns to element P on the top of FIG. 12 
where the process enters element 1202 which deter- 
mines that the mode status register 608 is set to the 
source mode and therefore outputs the process over 
path 1204 to element 1213. From there, since the backup 
register 623 is set, the process extends over path 1215 to 
element 1216. 

Let it be assumed for the time being that the access bit 
612 is not set to a 1 and therefore, the process extends 
over path 1218 to elemetn 1207 which causes the mode 
of the status register 608 to be set to the source mode. 
Since this register is already in this mode, no further 
action is taken by this element, The process then ex- 
tends to element 1209 which sets the mode TPN regis- 
ter 620 to a 1 and then outputs this 1 on all outgoing 
links of the node in element 1211. Element 1225 selects 



03/18/2004, EAST 



Version: 1.4.1 



4,736,393 

15 16 

the precision oscillator 505 so that it is in charge of the timing function. The purpose of the read TPN timer is 

timing as already described. The process then extends to limit the rate at which the reading operation of the 

over path 1212 then loops through the same sequence of TPN take place. The timer prevents this operation from 

elements as long as the access bit register 612 is not set. occurring to frequently by permitting it to occur only 

Finally, after a period of time the maintenance per- 5 when the timer times out. The process now extends 

sonnel will fix the problems in the master node and this over path 1613 to element 1614 which is also designated 

node will then once again take charge of the network. H within a circle. From there, the process continues to 

This is done by transmitting an appropriate message in element H on FIG. 17. 

one of the timeslots to this node. This message is entered On FIG. 17, the process extends from element H to 

into register 502 of system 501 and is transferred from 10 element 1701 which compares the A and B sections of 

there over path 604 where its contents are entered into the TPN register 616-619 pair of pairs having the low- 

the external access mode register 611 with the access bit est TPN. If a single A and B register pair has the lowest 

612 being set. The process then extends from element P TPN then only its A and B sections are compared. If 

on FIG. 12, through element 1202 to element 1213 two or more such pairs have the lowest TPN, then the 

where the backup register 623 is still set since the cur- 15 A and B section of each pair having the same lowest 

rently described mode is a backup master mode. From TPN is compared. 

there the process extends to element 1216. At this time, The process then extends over path 1702 to element 
the external access bit 612 is a 1 since a message has 1703 which determines by a comparison of the A and B 
been received by element 502 and entered into external section of the various one or more TPN register pairs 
access mode register 611. Therefore the process now 20 whether there has been any changes in the lowest TPN 
extends over path 1217 to elemetn 1219. Element 1219 recently received. If the answer is no, the process ex- 
now analyzes the message in the external access mode tends over path 1705 to element 1706 which determines 
register 611 to determine its contents. Since it is now whether the TPN stable timer 607 has been started. This 
desired that the master node resume control of the sys- timer is normally in a timed out state corresponding to 
tern, the message presently in register 611 specifies that 25 the state of the system in which no change in the lowest 
the node is to once again operate as a link node. This TPN has occurred for a period greater than its timeout 
means that it is to receive tuning signals from other period. It is reset when a change is detected in the low- 
nodes and to cease operating as a master node. There- est received TPN. A new link cannot be selected as a 
. fore, the process now extends over path 1220 to element timing source until this timer times out. This timer 
A and there to FIG. 13 which enters the process already 30 thereby affectively limits the frequency at which new 
described. links can be selected as new timing sources. 

With reference to element 1219, if the message writ- If element 1706 determines that the TPN stable timer 

ten into register 611 specifies that it is to continue oper- 607 has not been started, the process extends to element 

ating as a source node, then the process extends over 1710 where the timer is started. If the answer is yes in 

path 1221 to element 1207 and the process continues 35 element 1706, the process extends over path 1707 to 

using elements 1209, 1211 and 1225, etc. as already element 1712 which determines whether or not this 

described. timer has timed out. Thus, both paths 1707 and 1711 

At this time, the master node is once again in charge extend the process to element 1712 which now deter- 

of the system, it is generating a TPN of one and the mines whether or not the TPN stable timer 607 is timed 

timing signals it transmits out over its links are the tim- 40 out 

• ing signals that control timing of all other network If the timer has timed out then the process extends 

nodes as already described. over path 1713 to element 1714 which determines 

Now let it be assumed with reference to FIG. 15 that whether or not all valid TPNs are at their maximum 

element 1504 determines that the node TPn is within an value. It has been priorly mentioned that the maximum 

acceptable value and so the process extends over path 45 value for a TPN is 30 and that a TPN of 30 represents 

1505 to element 1516 and from there to element G on a trouble condition in which the node may not be pres- 

FIG. 16. On FIG. 16, the process extends from element ently receiving valid timing signals. Therefore, element 

G to element 1601 which reads the contents of the TPN 714 determines whether or not a TPN of 30 is now 

A registers 616 through 619. The process then extends being received on all links. It analyzes the contents of 

to element 1602 which finds the TPN A register having 50 the A registers 616 through 619 to make this determina- 

the lowest TPN. The process then extends over path tion. If the answer is no, the process extends over path 

1615 to element 1603 which determines whether or not 1715 to element 1716. Element 1716 cuases the code of 

more than one A register has the same lowest TPN the priorly designated link with the lowest TPN (see 

value. If the answer is no then the process extends over FIG. 16) to be written into the A port of the timing 

path 1605 to elemetn 1606. In this element the micro- 55 source select register 602. This register controls the 

processor 610 operates internally to preliminarily desig- switch controller 508 to set timing source selector 

nate the link associated with the lowest TPN. This link switch 504 to the position associated with the desig- 

is to be the preferred timing source for the node. nated link having the lowest TPN. 

With reference to element 1603, if the answer is yes, The process now extends over path 1717 to element 

the process extends over path 1604 to element 1607 60 1718 which determines whether or not the contents of 

wherein the processor 610 operates internally to desig- the A section of the timing source select register 602 

nate one of the links having the lowest TPN as the link equals the contents in the B section of the register. If the 

that is to be the preferred timing source for the node. answer is yes, this means that a new link has not been 

The process then extends either over path 1608 from selected and the process continues over path 1719 to 

element 1606 or over path 1609 from element 1607 to 65 element 1720 which rests the PLL timer 606. This timer 

element 1610 which causes the read TPN timer 605 to is normally in a timed out state and only then can one 

be reset. The process then extends over path 1611 to read the clock maintenance interface register 603. This 

element 1612 which starts the read TPN timer and its register interacts with the phase lock loop 506 to deter- 
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mine whether or not the phase lock loop is currently With reference to element 1712 let it now be assumed 

responding to a valid timing signal so as to properly that the TPN stable timer 607 has not timed out. In this 

control the clock signal generator 507 to generate a case the process extends over path 1739 to element 1726 

valid timing signal for the node. This timer gives the and to the processes already described. In this case, the 

phase lock loop 506 time to lock onto a new signal. This 5 TPN A registers 616 to 619 containing the lowest TPN 

timer also gets reset after there is a change with respect are stable, but the TPN stable timer 607 has not timed 

to the link that is to control the timing of the node. out, and a new link is not selected as a timing reference 

The process now extends over path 1721 to element because there is till a high probability of further TPN 

1722 which starts the PLL timer 606. The process then changes. 

extends over path 1723 to element 1724 which transfers 10 w * tfl respect to element 1714 let it be assumed that all 

the contents of the A section of the TPN registers 616 TPNs are at their maximum value. The process contin- 

through 619 to the B section of these registers. ues over P ath i™} to element 1740 which enters the 

The process continues over path 1725 to element node °f tne precision oscillator into the timing source 
1726 which causes the contents of the read TPN timer register 602. The process then continues over path 1742 
605 to be read. The timer is read and then the process 15 t0 tfle dements beginning with 1726 whose function has 
extends over path 1726A to element 1728 which deter- already been described. In this case all valid TPNs in 
mines whether or not the timer has timed out If it tnc TPN A re S ist crs 616 to 619 are at the maximum 
hasn't, the process loops over path 1727 and element there could be no valid source of timing from any 
1726 and path 1726A and then once again 1728 until the Iink since thcre ™ no operating master node in the net- 
timer does time out This timer prevents the system 20 wor k. Therefore the node timing will be generated from 
from reading the TPNs too often. the P rccsion oscillator until this condition ceases. 

Finally, a determination is made that the timer has , With to element 1718 now let it be assumed 

timed out and the process continues over path 1729 to that 111(5 contents of the A section of the timing source 

element 1730 which moves the contents of the TPN A sele ? t re f ister 602 do not match the contents of the B 

registers to the TPN B registers. These are the register 25 s f t } 0Q . of th J s therebv bating that the source 

pairs 616 through 619. The process then continues over °* tmu \ s T of ! he phase } ock * d Io0 P 506 has J ust bccn 

path 1731 to element F and from there back to element d \ In tlu ' c ^ continues over path 

F on FIG, 15 where the processes already described are ?™J° T el ^ me nt ™* which determines whether or not 

entered L timer ^ nas timed out - If u 1S not timed out » 

Briefly, the process of FIG. 15 outputs the current 30 ? C re S^r 603 cannot be read 

node TPN if the node TPN is not at its^um vZ ^^^S^^^^ ^ 

It causes the node to become a source node if the node ZTm? ^ ^ 

t-cixt * * i * , . , . . sen bed. When the PLL timer 606 times out, this indi- 

^iZ^ " deS,gBated «*» *° P hase locked l0 °P 506 has connect* 

^ P masiernoac - 35 to an input signal sufficiently long enough to determine 

17^*^ tTn* ° n ^ * i-VrSig Properly and that the frequency of its 

TPN rl^l SSS^S A?Si^ hi extends over *> ath 1146 10 element 1747 which reads the 

J^^S^^f}^ u^^ n "^^ contents of the clock ^^enance register 603. The 

^SZ^TtST™? 'S^ 16 18 m l 40 contents of this "tf** are a 1 when the phase lock 

change detected in the owest received TPN as a result loo 506 b ratin ri and 0 when it is not 

f h C t r P ^° n ^T 11 1701 'i f A^ ange 1 " dC " °P eratin « P r °P erl y- If elcmcnt ™* determines that this 

™ ' I *\ P 7i eXtends k °r Path 17 °t t0 Clement re S* ter ' s contents « a L th <* the process continues 

1732 which determines whether or not the precision over path 1750 t0 elemcnt m4 whose ftinction has 

45 already been described. This indicates that the phase 
ter 602 to adjust switch 504 If 'the answer is yes, the locked loop 506 is oper ating properly and is locked to 
process continues over path 1733 and the TPN stable henC e deriving timing from its input signal from 
timer 607 is reset m element 1737. If the answer of ele- timing source selector switch 504. Hence no corrective 
ment 1732 is no, then the precision oscillator code is action is required. On the other hand, if element 1749 
entered mto the A portion of the timing source select 50 determines that the clock maintenance register contains 
register 602 : and I the process continues over path 1736 to a 0, this means that the phase lock loop 506 is not recei v- 
element 1737. The resetting of the TPN stable timer ing a goo d timing signal. In this case, the process ex- 
prevents a new link from being selected until this timer tends over path 1751 to element 1752 which causes the 
tunes out The process now continues over path 1738 to precision oscillator code to be entered into the A sec- 
element 1726 which reads the contents of the read TPN 5 5 tion of the timing source select register 602. This pre- 
timer 605 and then element 1728 determines whether or pares the node to operate under control of its precision 
not the read TPN timer 605 is timed out. The function oscillator 505. The process then extends over path 1753 
of element 1730 has already been described. to element 1754 which writes an out of service bit of 1 
Thus, in partial summary, the process just described next to the TPN of the selected link in the A section of 
beginning with elements 1703 and 1732 caused ten pre- 60 the TPN register 616 through 619 associated with the 
cision oscillator to be selected. A new link will be se- selected link. This out of service bit of 1 in this register 
lected as the nodes source of timing after the TPN indicates that the phase lock loop 506 is having diffi- 
fluctuations cease and the ssytem returns to a stable culty following the timing signal provided by the se- 
state as indicated by steady TPNs. However, a new link lected link. This determination was made when it was 
cannot be selected until after the TPN stable timer times 65 determined that the clock maintenance register 603 
out and we can be relatively certain that there will be no contains a 0 rather than 1. 

more fluctuations in the TPN A registers 616 to 619 A 0 in register 603 indicates that the phase lock loop 

containing the lowest TPN. 506 cannot successfully follow the timing signal to 
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which the phase lock loop is currently connected by 
switch 504. That is the reason that element 1752 entered 
the precision oscillator code in the timing source select 
register 602 to cause the swtich 504 to connect the 
phase lock loop 506 to the output of the precision oscil- 5 
lator 505 and to disconnect it from the link providing 
the unreliable timing signal. Therefore, element 1754 
writes an out of service bit of 1 into the A section of the 
TPN registers associated with this link. This out of 
service bit of 1 advises the processor 610 that this TPN 10 
for this link in this register should be disregarded until 
the system is reinitialized starting at element 1101 of 
FIG. 11. The reason for doing this is to prevent the 
clock system from attempting to synchornize to this 
defective link on subsequent reading of the TPNs. This 13 
condition persists until the out of service bit for this 
register is switched from a 1 to a 0 to indicate that the 
associated link may once again by providing a reliable 
timing signal. 

The process next continues from element 1754 and 20 
over path 1755 to element 1756. This element causes an 
entry to be made in alarm register 614 to activate the 
alarm LED's 613. These provide an alarm to the main- 
tenance personnel indicating that one of the incoming 
links is providing an unreliable timing signal. The main- 
tenance personnel then take the appropriate action re- 
quired to remedy the situation. 

The process next advances over path 1757 to element 
1724. The element causes the current contents of the A 
section of the timing source select register written into 
the B section of the register. 

The process next advances over path 1725 to element 
605 and the elements subsequent thereto. This portion 
of the flow chart has already been described. 35 

On FIG. 6, conductors 511, 510, 604, 331, 325, 335, 
425, 435 and 332, respectively are shown as being di- 
reclty connected to registers 603, 602, 611, 615, 616, 
617, 618, 619, and 620, respectively for ease of under- 
standing of the relationship between these conductors ^ 
and registers. In practice, since node controller 601 is 
processor controlled, these registers would be part of 
the memory of microprocessor 610 and these conduc- 
tors would be connected to I/O facilities of the micro- 
processor. 45 

While a specific embodiment of the invention has 
been disclosed, variations within the scope of the ap- 
pended claims are possible and are contemplated. There 
is no intention of limitation to what is contained in the 
abstract or the exact disclosure as herein presented. The 50 
above-described arrangements are only illustrative of 
the application of the principles of the invention. It is to 
be expected that other arrangements will be devised by 
those skilled in the art without departing from the spirit 
and the scope of the invention. In particular, an embodi- 55 
ment where the precision oscillator is replaced by a 
combination of a precision oscillator and one or more 
timing reference signals of origin outside the presently 
described network of nodes is contemplated. 

What is claimed is: go 

1. A method of controlling the timing of each node in 
a multinode system whose nodes are interconnected by 
links and wherein a clock source in one of said nodes 
provides a reference clock signal source for controlling 
the tuning of all other nodes in said system, said method 65 
comprising the steps of: 

(1) operating one of said nodes as a master node to 
control the timing of all nodes of said system, 



(2) transmitting a clock signal between each pair of 
said nodes directly interconnected by any one of 
said links, 

(3) including timing priority number (TPN) informa- 
tion in the clock signal transmitted between each 
pair of directly connected nodes with said informa- 
tion representing the number of intermediate nodes 
said clock signal has encountered in traveling from 
said master node to the node receiving said clock 
signal, 

(4) operating each clock signal receiving node, other 
than said master node, to analyze the TPN informa- 
tion it receives on all links to which it is connected 
to identify as a preferred clock signal the clock 
signal that has encountered the least number of 
intermediate nodes in traveling from said master 
node to reach said clock signal receiving node, 

(5) modifying the received TPN from the intermedi- 
ate node generating said preferred clock signal, 

(6) transmitting the modified TPN to all intermediate 
nodes, and 

(7) operating each clock signal receiving node, other 
than said master node, to select said preferred 
clock signal as a reference timing source for said 
receiving node. 

2. The method of claim 1 in combination with the 
additional steps of: 

(1) operating each node, other than said master node, 
to repeatedly scan said TPN information it re- 
ceives, 

(2) detecting when the received TPN information 
associated with said preferred clock signal changes 
on successive scans, and 

(3) switching the timing control of each node that 
detects a change from said preferred clock signal to 
a signal from a local oscillator internal to said last 
named node upon the detection of said change. 

3. The method of claim 2 in combination with the 
additional steps of: 

(1) operating said last named node to continue its 
repeated scanning of said received TPN informa- 
tion, 

(2) comparing the results of successive scans, 

(3) determining when said received TPN information 
remains unchanged for a predetermined interval of 
time, and 

(4) analyzing said received TPN information after the 
expiration of said interval to select as a preferred 
clock signal for said node the signal that has en- 
countered the least number of intermediate nodes 
in traveling from said master node to said receiving 
node. 

4. The method of claim 1 in combination with the 
additional steps of: 

(1) operating each node, other than said master node, 
to repeatedly scan said TPN information it re- 
ceives, 

(2) operating one of said other nodes as a backup 
master node, 

(3) operating said backup master node to detect when 
said received TPN information associated with 
said preferred clock signal changes from a prior 
value to a predetermined value, 

(4) switching the timing control of said backup mas- 
ter node from said preferred received clock signal 
to a signal from a local oscillator internal to said 
backup master node upon the detection of said 
change to said predetermined value, and 
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(5) transmitting a clock signal containing TPN infor- 
mation over directly connected ones of said links 
from said backup master node to control the timing 
of all nodes of said system other than said master 
node. 5 

5. The method of controlling the timing of each node 
in a multinode system whose nodes are interconnected 
by links and wherein a clock source in one of said nodes 
provides a reference clock signal source for all other 
nodes in said system, said method comprising the steps 10 
of: 

(1) operating one of said nodes as a master node to 
control the timing of all nodes of said system, 

(2) including a timing priority number (TPN) in a 
clock signal transmitted from said master node 15 
over a direct link to at least another one of said 
nodes,- 

(3) operating each node, other than master node, so 
that it alters a TPN it receives over a directly con- 
nected link in a prescribed manner to form a new 20 
TPN, 

(4) further operating each node, other than said mas- 
ter node, so that each said node sends a clock signal 
together with its altered TPN to all nodes to which 

it is directly connected by a link, 25 

(5) operating each node, other than said master node 
to identify one of the TPNs received as a preferred 
TPN based upon a prescribed relationship of said 
received TPNs, and 

(6) controlling the timing of said nodes, other than 30 
said master node, by the clock signal received over 
the link over which said preferred TPN is received 
by said node. 

6. The method of claim 5 in combination with the 
additional steps of: 35 

(1) operating each node, other than said master node, 
to repeatedly scan the TPNs it receives, 

(2) detecting a change when the received TPNs asso- 
ciated with said preferred clock signal changes on 
successive scans, and 40 

(3) switching the timing control of said node that 
receives said changed TPNs from said preferred 
clock signal to a signal from a local oscillator inter- 
nal to said last named node upon the detection of 
said change. 45 

7. The method of claim 6 in combination with the 
additional steps of; 

(1) operating said last named node to continue its 
repeated scanning of said received TPNs, 

(2) comparing the results of successive scans, 50 

(3) determining when said received TPNs remain 
unchanged for a predetermined interval of time, 
and 

(4) analyzing said received TPN information after the 
expiration of said interval to select as a preferred 55 
clock signal for said node the clock signal received 
over the link on which a TPN having a prescribed 
relationship to the TPNs of other links is received. 

8. The method of claim 5 in combination with the 
additional steps of: 60 

(1) operating each node, other than said master node, 
to repeatedly scan said received TPNs, 

(2) operating one of said other nodes as a backup 
master node, 

(3) operating said backup master node to detect when 65 
said received TPN associated with said preferred 
clock signal changes from a prior value to a prede- 
termined value, and 
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(4) switching the timing control of said backup mas- 
ter node form said preferred received clock signal 
to a signal from a local oscillator internal to said 
backup master node upon the detection of said 
change to said predetermined value, and 

(5) transmitting a clock signal contain a TPN over 
directly connected ones of said links from said 
backup master node to control the timing of all 
nodes of said system other than said master node. 

9. A method of controlling the timing of a clock 
signal receiving node in a multinode system whose 
nodes are interconnected by links and wherein a clock 
source in a master one of said nodes provides a refer- 
ence clock signal source for controlling the timing of all 
other nodes in said system, said method comprising the 
steps of: 

(1) receiving timing priority number (TPN) informa- 
tion together with a clock signal on each different 
link connected to said clock signal receiving node 
with said information representing the number of 
intermediate nodes each received clock signal has 
encountered in traveling from said master node to 
said clock signal receiving node, 

(2) operating said clock signal receiving node to ana- 
lyze the TPN information it receives on all links to 
which it is connected to identify as a preferred 
clock signal the clock signal that has encountered 
the least number of intermediate nodes in traveling 
from said master node to reach said node, 

(3) modifying the TPN from the link from which said 
preferred clock signal was selected, 

(4) transmitting the modified TPN on all links to 
which said clock signal receiving node is con- 
nected, and 

(5) operating said signal receiving node to select said 
preferred clock signal as a reference timing source 
for said signal receiving node. 

10. A method of controlling the timing of each node 
in a multinode system wherein said nodes are connected 
by links and wherein a clock source in a master one of 
said nodes provides a reference clock signal for all other 
nodes in said system, said method comprising the steps 
of: 

(1) interconnecting each of said nodes to one or more 
other ones of said nodes via said links, 

(2) transmitting a clock signal between each of said 
nodes directly interconnected by any one of said 
links, 

(3) activating one of said nodes to operate as a master 
node by providing a reference clock signal for 
controlling the timing of said system, 

(4) including a timing priority number (TPN) of n in 
the clock signal sent from said master node to all of 
said nodes directly connected to said master node 
by any one said links, 

(5) operating each node directly connected by a link 
to said master node and receiving said TPN of n to 
alter n by m to form a TPN of n+m characterizing 
each said directly connected receiving node, 

(6) further operating each node having a TPN of 
n+m to include said TPN of n-f m in the clock 
signal transmitted to each one of said nodes to 
which said node having a TPN of n-hm is directly 
connected by any one of said links, 

(7) operating all others of said nodes, other than said 
master node, that receives a TPN of n-hm so that 
each of said other nodes alters a TPN it receives by 
m and transmits its altered TPN with a clock signal 
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to each node to which it is directly connected by 
any one of said links, 

(8) operating each node that receives more than a 
single TPN, other than said master node, to select 
one of said received TPNs as a preferred received 5 
TPN, and 

(9) controlling the timing of each node, other than 
said master node, from clock signals transmitted 
over the link that applies the preferred TPN to said 
controlled node. 10 

11. The method of claim 10 in combination with the 
additional steps of: 

(1) operating each node, other than said master node, 
to repeatedly scan the TPNs it receives, 

(2) detecting a change when the received TPN asso- 15 
dated with said preferred clock signal changes on 
successive scans, and 

(3) switching the timing control of said node that 
detects a changed TPN from said preferred clock 
signal to a signal from a local oscillator internal to 20 
said last named node upon the detection of said 
change. 

12. The method of claim 11 in combination with the 
additional steps of: 

(1) operating said last named node to continue its 25 
repeated scanning of said received TPNs, 

(2) comparing the results of successive scans, 

(3) determining when said received TPNs remain 
unchanged for a predetermined interval of time, 
and 30 

(4) analyzing said received TPNs after said interval to 
select as a preferred clock signal for said node the 
clock signal received over the link on which a TPN 
having a prescribed relationship to the TPNs of 
other links is received. 35 

13. The method of claim 10 in combination with the 
additional steps of: 

(1) operating each node, other than said master node, 
to repeatedly scan said received TPNs, 

(2) operating one of said other nodes as a backup 40 
master node, 

(3) operating said backup master node to detect when 
said received TPN associated with said preferred 
clock signal changes from a prior value to a prede- 
termined value, and 45 

(4) switching the timing control of said backup mas- 
ter node from said preferred received clock signal 
to a signal from a local oscillator internal to said 
backup master node upon the detection of said 
change to said predetermined value, and 50 

(5) transmitting a clock signal containing a TPN over 
directly connected ones of said links from said 
backup master node to control the timing of all 
nodes of said system other than said master node. 

14. Facilities for controlling the timing of each node 55 
in a multinode system whose nodes are interconnected 
by links and wherein a clock source in one of said nodes 
provides a reference clock signal source for controlling 
the timing of all other nodes in said system, said facili- 
ties comprising: 60 

means for operating one of said nodes as a master 
node to control the timing of all nodes of said sys- 
tem, 

means for transmitting a clock signal between each 
pair of said nodes directly interconnected by any 65 
one of said links, 

means for including timing priority number (TPN) 
information in the clock signal transmitted between 



each pair of directly connected nodes with said 
information representing the priority of said clock 
signal, 

means for operating each clock signal receiving node, 
other than said master node, to analyze the TPN 
information it receives on all links to which it is 
connected to identify a preferred clock signal, 

means for modifying the TPN from the directly con- 
nected node which transmitted said preferred 
clock signal, 

means for transporting the modified TPN to all nodes 
directly connected to said clock signal receiving 
node, and 

means for operating each clock signal receiving node 
to select said preferred clock signal as a reference 
timing source for said receiving node. 

15. The facilities of claim 14 in combination with: 
means for operating each node, other than said master 

node, to repeatedly scan said TPN information it 
receives, 

means for detecting when the received TPN informa- 
tion associated with said preferred clock signal 
changes on successive scans, and 

means for switching the timing control of said node 
that receives said changed TPN information from 
said preferred clock signal to a signal from a local 
oscillator internal to said last named node upon the 
detection of said change. 

16. The facilities of claim IS in combination with: 
means for operating said last named node to continue 

its repeated scanning of said received TPN infor- 
mation, 

means for comparing the results of successive scans, 
means for determining when said received TPN in- 
formation remains unchanged for a predetermined 
interval of time, and 
means for analyzing said received TPN information 
after the expiration of said interval to select as a 
preferred clock signal for said node the signal that 
has encountered the least number of intermediate 
nodes in traveling from said master node to said 
receiving node. 

17. The facilities of claim 14 in combination with: 
means for operating each node, other than said master 

node, to repeatedly scan said TPN information it 
receives, 

means for operating one of said other nodes as a 
backup master node, 

means for operating said backup master node to de- 
tect when said received TPN information associ- 
ated with said preferred clock signal changes from 
a prior value to a predetermined value, 

means for switching the timing control of said backup 
master node from said preferred received clock 
signal to a signal from a local oscillator internal to 
said backup master node upon the detection of said 
change to said predetermined value, and 

means for transmitting a clock signal containing TPN 
information over directly connected ones of said 
links from said backup master node to control the 
timing of all nodes of said system other than said 
master node. 

18. Facilities for controlling the timing of each node 
in a multinode system whose nodes are interconnected 
by links and wherein a clock source in one of said nodes 
provides a reference clock signal source for all other 
nodes in said system, said facilities comprising: 
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means for operating one of said nodes as a master 
node to control the timing of all nodes of said sys- 
tem, 

means for including a timing priority number (TPN) 
in the clock signal transmitted from said master 5 
node over a direct link to at least another one of 
said nodes, 

means for operating each node, other than master 
node, so that it alters a TPN it receives over a 
directly connected link in a prescribed manner to 10 
form a new TPN, 

means for further operating each node, other than 
said master node, so that each said node sends a 
clock signal together with its altered TPN to all 
nodes to which it is directly connected by a link, 15 

means for operating each node, other than said master 
node, to identify one of the TPNs received as a 
preferred TPN based upon a prescribed relation- 
ship of said received TPNs, and 

means for controlling the timing of each said node, 20 
other than said master node, by the clock signal 
received over he link over which said preferred 
TPN is received by said node. 

19. The facilities of claim 18 in combination with: 
means for operating each node, other than said master 25 

node, to repeatedly scan said TPNs it receives, 
means for detecting a change when the received TPN 
associated with said preferred clock signal changes 
on successive scans, and 
means for switching the timing control of said node 30 
that receives a changed TPN from said preferred 
clock signal to a signal from a local oscillator inter* 
nal to said node upon the detection of said change. 

20. The facilities of claim 19 in combination with: 
means for operating said last named node to continue 35 

its repeated scanning of said received TPNs, 
means for comparing the results of successive scans, 
means for determining when said received TPNs 
remain unchanged for a predetermined interval of 
time, and 40 
means for analyzing said received TPNs to select as a v 
preferred clock signal for said node the clock signal 
received over the link on which a TPN having a 
prescribed relationship to the TPNs of other links 
is received. 45 

21. The facilities of claim 18 in combination with: 
means for operating each node, other than said master 

node, to repeatedly scan said received TPNs, 

means for operating one of said other nodes as a 
backup master node, 50 

means for operating said backup master node to re- 
peatedly scan said received TPNs to detect when 
said received TPN associated with said preferred 
clock signal changes from a prior value to a prede- 
termined value, and 55 

means for switching the timing control of said backup 
master node from said preferred received clock 
signal to a signal from a local oscillator internal to 
said backup master node upon the detection of said 
change to said predetermined value, and 60 

means for transmitting a clock signal containing a 
TPN over directly connected ones of said links 
from said backup master node to control the timing 
of all nodes of said system other than said master 
node. 65 

22. Facilities for controlling the timing of each node 
in a multinode system wherein said nodes are connected 
by links and wherein a clock source in a master one of 



said nodes provides a reference clock signal for all other 
nodes in said system, said facilities comprising: 
links for directly interconnecting each of said nodes 

to one or more other of said nodes, 
means for transmitting a clock signal between each of 
said nodes directly interconnected by any one of 
said links, 

means for activating one of said nodes to operate as a 
master node by providing a reference clock signal . 
for controlling the timing of said system, 

means for including a timing priority number (TPN) 
of n in the clock signal sent from said master node 
to all of said nodes directly connected to said mas- 
ter node by any one of said links, 

means for operating each node directly connected by 
a link to said master node and receiving said TPN 
of n to alter n by m to form a TPN of n+m charac- 
terizing each said directly connected receiving 
node, 

means for further operating each node having a TPN 
of n-f m to include said TPN of n+m in the clock 
signal transmitted to each one of said nodes to 
which said node having a TPN of n-f m is directly 
connected by any one of said links, 

means for operating all others of said nodes, other 
than said master node, that receives a TPN of n+m 
so that each of sad other node alters a TPN it re- 
ceives by m and transmits its altered TPN with a 
clock signal to each node to which it is directly 
interconnected by any one of said links, 

means for operating each node that receives more 
than a single TPN, other than said master node, to 
select one of said received TPNs as a preferred 
received TPN, and 

means for controlling the timing of each node, other 
than said master node, from clock signals transmit- 
ted over the link that applies the preferred TPN to 
said controlled node. 

23. Facilities for controlling the timing of a clock 
signal receiving node in a multinode system whose 
nodes are interconnected by links and wherein a clock 
source in a master one of said nodes provides a refer- 
ence clock signal source for controlling the timing of all 
other nodes in said system, said facilities comprising: 

means for receiving timing priority number (TPN) 
information together with a clock signal on each 
different link connected to said signal receiving 
node with said TPN information representing the 
number of intermediate nodes each received clock 
signal has encountered in traveling from said mas- 
ter node to said signal receiving node, 

means for operating said signal receiving node to 
analyze the TPN information it receives on all links 
to which it is connected to identify as a preferred 
clock signal the clock signal that has encountered 
the least number of intermediate nodes in traveling 
from said master node to reach said signal receiv- 
ing node, 

means for selecting said preferred clock, signal as a 
reference timing source of said signal receiving 
node, 

means for modifying the TPN received from the link 
from which said preferred clock signal was re- 
ceived, and 

means for transmitting the modified TPN on all links 
to which said signal receiving node is connected. 

24. Facilities for controlling the timing of a clock 
signal receiving node in a multinode system whose 
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nodes are interconnected by links said facilities compris- 
ing: 

means for receiving timing priority number (TPN) 
information together with a clock signal on each 
different link connected to said signal receiving 
node with said TPN information representing the 
relative priority of each received clock signal, 

means for operating said signal receiving node to 
analyze the TPN information it receives on all links 
to which it is connected to identify a preferred 
clock signal having a preferred priority, 

means for selecting said preferred clock signal as a 
reference timing source for said signal receiving 
node, 

means for modifying the TPN received from the link 
from which said preferred clock signal was re- 
ceived, and 

means for transmitting the modified TPN on all links 
to which said signal receiving node is connected. 

25. A method of controlling the timing of each nodes 
in a multinode system whose nodes are interconnected 25 
by links and wherein a clock source in one of said nodes 
provides a reference clock signal source for controlling 
the timing of all other nodes in said system, said method 
comprising the steps of: 

(1) operating one of said nodes as a master node to 
control the timing of all nodes of said system, 
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(2) transmitting a clock signal between each pair of 
said nodes directly interconnected by any one of 
said links, 

(3) including timing priority number (TPN) informa- 
tion in the clock signal transmitted between each 
pair of directly connected nodes with said informa- 
tion representing the relative priority of the clock 
signal used by each transmitting node, 

(4) ojperating each signal receiving node, other than 
said master node, to analyze the TPN information 
it receives on all links to which it is connected to 
identify a preferred clock signal having a preferred 
priority, and 

(5) operating each clock signal receiving node, other 
than said master node, to select said preferred 
clock signal as a reference timing source for said 
receiving node, 

(6) modifying the TPN received from the link from 
which said preferred clock signal was received, 
and 

(7) transmitting the modified TPN on all links to 
which said receiving node is connected. 

26. The method of claim 25 in combination with the 
additional steps of; 

(1) detecting when the received TPN information 
associated with said preferred clock signal 
changes, and 

(2) switching the timing control of each node that 
detects a change from said preferred clock signal to 
another clock signal upon the detection of said 
change. 
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